Virtual reality driver training and assessment system

ABSTRACT

A virtual reality (VR) based driving simulation system and method provides hazard perception training and testing. Some embodiments use a personal computing device (e.g., smartphone), a head-mounted VR headset, and a Bluetooth-connected push button keypad or other user input device. An immersive, 3D simulator enables users to experience and learn from hazardous scenarios without risking injury. Performance by a user in perceiving a hazard is compiled as a metric in a user profile stored on a server, and continued training and testing can be conducted with updated scenarios supplied by the server in accordance with the metric associated with a user.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application Ser.No. 62/601,690, filed Mar. 28, 2017, which is incorporated herein byreference.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH

Not Applicable.

BACKGROUND OF THE INVENTION

The present invention relates in general to driving simulators, and,more specifically, to a virtual reality (VR) driving simulator with atraining mode and an assessment mode.

Training of young or novice drivers is an important measure to reducemotor vehicle crashes. Hazard perception/anticipation performance,defined as a driver's ability to detect and anticipate dangerous drivingsituations on the road, is an effective predictor of crash risk andon-road driving proficiency. Before full licensing is granted, youngdrivers usually receive driving education from either adult licenseddrivers (e.g., parents or close friends) or from professionalinstructors, or both. However, neither of the two commonly adoptededucation approaches has the ability to provide enough exposures tohazardous driving situations. In order to get a driver's license, novicedrivers are required to understand traffic rules and to demonstrate thatthey can drive in various environments and successfully perform variousparking maneuvers. However, most, if not all, of the training andtesting are carried out under optimum conditions in which crash hazardsare rare. Supplemental training experiences to train and assess novicedrivers on their hazard perception and anticipation skills would bebeneficial to improve handling of hazardous situations that inevitablyarise once regular driving begins.

Some training tools have been developed aiming to provide interventionfor the hazard perception issue for younger drivers. They are typicallybased on still images or video footage depicting representative drivingscenarios. Participants in these programs are often required to reportor use mouse clicks to identify the potential hazards or hazardous area.While helpful, such training has not become widely used, in part becausethere has been a lack of ecological validity in representing real worldsituations, a lack of engagement/interest from potential students, and alack of a reward system.

SUMMARY OF THE INVENTION

The invention provides an immersive driving environment including manyreal-world hazardous scenarios presented in a manner that trains andthen assesses users' situational awareness of crash hazards in arealistic driving environment. Potentially hazardous scenarios caninclude left turns, approaching emergency vehicles, constructionimpediments, disabled vehicles, and many other scenarios with thepotential to result in collisions with vehicles, cyclists, pedestrians,and fixed obstacles. The inventive driving simulator can be similar inpresentation to a video game in order to increase a user's engagementwith the simulator. In addition to improving a young driver's skills inanticipating and reacting to potential hazards, the invention can alsoprovide an objective assessment of a user's overall accident risk whichcould be useful for public agencies or insurance companies.

In one particular aspect of the invention, a virtual reality (VR)driving simulator for presenting simulated hazard events to a usercomprises a VR headset mountable to a head of the user. The VR headsethas a display system for presenting respective left and right images toleft and right eyes of the user. A simulation controller is coupled tothe display system for generating 3D animations simulating a drivingsequence. A user control device is wirelessly coupled to the simulationcontroller for generating at least one command signal corresponding to adriving action in response to the 3D animations. The 3D animationsgenerated by the simulation controller include a path progressionsequence depicting a vehicle drive cycle including driver actionsdefined by the command signal. The 3D animations generated by thesimulation controller also include a hazard scene introduced into thepath progression sequence at a predetermined moment representing asafety risk which is dependent upon a user perception and a userreaction. The simulation controller compares the user perception topredetermined perception performance levels. The simulation controllerhas a training mode wherein when the a user perception is detected belowa selected one of the performance levels then the hazard scene includesan instruction phase highlighting a source of the safety risk. Thesimulation controller also has a testing mode wherein the instructionphase is not included.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram showing one preferred embodiment of the inventionusing a head-mounted VR display, smartphone, and handheld controller.

FIG. 2 depicts an animated scene as shown on a VR display.

FIG. 3 is a flowchart showing one preferred embodiment for a drivingsimulation session for training and assessing an ability to detectdriving hazards.

FIG. 4 is a block diagram showing one preferred system architecture ofthe invention.

FIG. 5 is a top view of a user input device.

FIG. 6 is a block diagram showing a preferred embodiment of theinvention having a VR system using a personal mobile device such as asmartphone.

FIG. 7 is a flowchart showing a method of the invention in greaterdetail.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

The invention provides a driving simulation system based on a VRheadset, a computing device (e.g., personal computer, smartphone, ortablet), a manual input device (i.e., game controller), and a networkserver. The smartphone provides the main simulation controller executingan application program that handles the visual 3D displays, takes inputsfrom the input device, and processes the input and makes decisions forselecting and implementing training/testing scenarios. The VR headset isput on the head of the user, and displays an immersive 3D environment.The network server collects data from the smartphone and sends commandsand scenario/hazard data to the smartphone for customizing/updating thescenarios.

The input device used for the hazard perception assessment/training canbe a typical Bluetooth handheld device (e.g., similar to a gamingcontroller) that is sometimes supplied with a VR headset product (e.g.,Google Daydream or Samsung Gear VR), or can be a specially designedsteering wheel with brake/accelerator pedal (of a type as used for knownNintendo video game systems). The application program simulates drivingbased on user-controlled parameters, which may be a subset of all thedriving control actions that a driver might use in the real world. Forexample, users may control the speed and moving direction of thesimulated vehicle by pressing action keys (UP-forward, DOWN-backward,LEFT-left turn, RIGHT-right) on a button keypad. The video/audio contentof the simulation preferably follows the user controls to virtuallynavigate through a 3D environment of streets, traffic signals, scenery,and other vehicles. At selected times, potentially hazardous scenes areintroduced into the simulation that cover real-world incidents thatraise a safety risk that needs to be perceived and acted upon by thedriver. Particular types of hazard events can be updated via a network(e.g., over the Internet) so that the hazards become more challenging ordifficult to detect or respond to as users make progress. The userscould choose different modes of testing which will present differenttravel paths instead of following a fixed path.

The successful detection of the hazard can be determined in severalways. For example, users can be instructed to slow down their vehiclewhenever they detect a hazard. After slowing down they may be directedto use a “point and click” action using the input device, or they cansimply turn their head to face the hazard to indicate a successfuldetection of a hazard. For a headset with eye-tracking functions, theuser's fixation would be used to detect perception of a hazard. Afterthat, the user may be instructed to wait until the hazard disappears(e.g., a pedestrian finishes walking across the street or changeshis/her moving direction and becomes a non-threat) and then proceed todrive under normal training mode. If the user fails to detect the hazardwhile in a training mode, the vehicle may preferably stop automaticallyin front of the hazard so that the user can be shown where the hazardcame from before they proceed. The user can be given the option ofchoosing to replay the hazardous scenario from a different perspective(e.g., a bird's eye view or a top view). Under testing mode, theinstruction scene will not be presented. Instead, a score may berecorded and the missed hazardous scenarios will be re-presented to theuser at the end of the test.

A VR-headset-based embodiment is shown in FIG. 1. A user 10 is wearing aVR headset 11. In a standalone system, headset 11 may incorporate dualdisplays and a processor containing appropriate hardware and softwarefor executing a training/testing system as described herein. In asmartphone system, headset 11 accepts a smartphone 12 in a compartment13. Smartphone 12 or other personal mobile device provides the necessarydisplay and computing resources. In a computer-based system, headset 11would accepts video and audio signal from a computing device such as agaming computer (not shown), either via cables or wirelessly. In such anembodiment, the computing device would provide the necessary display andcomputing resources. In any case, a handheld, wireless input device 14(e.g., a button pad or other gaming-type device) provides manual inputsincluding direction buttons 15 and a select or enter button 16.Direction buttons 15 (e.g., Left, Right, Up, and Down) can be used tocontrol a virtual speed and direction of a simulated vehicle during adriving sequence and can be used at other times to selectably highlightmenu items or point to different objects, while select button 16 is usedto confirm a selection. A double click of select button 16 can be usedto move a test to the next trial or scenario. Smartphone 12 or astandalone VR headset 11 can be wirelessly coupled to a network server(not shown) which collects user performance data from the smartphone andprovides customized scenarios to be used in the simulation applicationfor adjusting the test or training parameters for a particular user. ABluetooth connection may also be provided between smartphone 12 andheadphones 17 which can be used to provide auditory feedback or promptsto user 10.

FIG. 2 shows a frame 20 from an animation that unfolds according to asimulated driving sequence that follows a drive cycle under partialcontrol of the user. A 3D display system is used to present respectiveleft and right images to left and right eyes of the user in order tocreate an immersive environment to actively engage the user's attentionand interest. The environment may represent a vehicle interior 21, astreet 22, and landscape 23, for example. The 3D animations for thesimulation include a path progression sequence such that the vehicledrive cycle may follow a route along predefined streets in the virtualworld selected by driver actions entered on the manual input device tosupply various command signals to the simulation controller.

At a moment selected by the simulation controller, the 3D animationsgenerated by the simulation controller introduce a hazard scene into thesequence representing a safety risk. In particular, a virtual object maybe introduced with a relative location and trajectory corresponding to apotential collision between the object and the simulated vehicle. Forexample, a pedestrian 24 is shown in FIG. 2 walking onto street 22 infront of the user which represents a safety risk wherein the amount ofrisk depends on the user's quick perception/recognition of the hazardand the driving actions taken in response.

FIG. 3 shows a preferred operation for the simulator system wherein auser (i.e., a target Learner such as a novice driver) opens theapplication program in step 25. Using the preferred hardware system, theuser will have set up their mobile device within a VR headset and pairedthe mobile device with a manual input device. In step 26, the Learner“drives” a simulated vehicle along a driving sequence. Periodicallyduring the simulation, realistic hazards events or scenes are presentedin step 27. Beginning at the moment of presenting a hazard, theapplication program monitors for a perception of and/or a reaction tothe hazard by the Learner. A determination is made in step 28 whetherthe Learner successfully detected the hazard. Successful detection maybe comprised of the Learner demonstrating a predetermined perceptionperformance level. The performance levels can be binary (e.g., anability to notice an anomaly and characterize it as a hazard within xseconds or not) or can be multi-level.

In the event that the Learner does not successfully detect the hazard instep 28, then an in-game point score maintained for the Learner isdecremented in step 29. Assigning a point score to the Learner providesmotivation for devoting full effort toward the task of detecting thehazards. For further motivation, a gamified sound feedback effect (e.g.,a stinger sound effect with a negative connotation) is played in step30. The Learner's online performance tracker (i.e., profile) is updatedin step 31. To reinforce the correct behavior, the simulation of thedrive cycle is continued in step 32 with the same hazard scene beingpresented again to the Learner.

When the Learner successfully detects the hazard by satisfying thepredetermined performance level, their in-game point score isincremented in step 33. A gamified sound feedback effect with a positiveconnotation is played in step 34 and the online performance tracker isupdated in step 35. The Learner carries on with the driving simulationwith step 26.

A hardware architecture for practicing the foregoing method is shown inFIG. 4. Whether implemented using a smartphone or other platform forexecuting a corresponding application program, a simulation controller40 is configured to drive a VR display 41 in a VR headset. Simulationcontroller 40 is preferably coupled to headphones 42 for providinginstructions, performance feedback, and other information to user 10. Auser input device includes a pointer 44 and clicker 45 which supply theuser's manual input to simulation controller 40. Block 43 representssensors and actuators also used by simulation controller to interfacewith user 10, including motion sensors (e.g., accelerometers orgyroscopes) to monitor head movements and vibration motors to generatehaptic feedback.

Simulation controller 40 preferably is comprised of a control block 46,a judgment block 47, a decision block 48, and a display block 49.Control block 46 controls the overall organization and operation of theapplication trials and the scoring functions, for example. Judgmentblock 47 evaluates user input to determine whether detection of a hazardis timely or not. Judgment block 47 may generate auditory feedback to bepresented to the user via headphones 42 to inform the user of theoccurrence of errors. Display block 49 handles the creation andanimation of the 3D environment, objects, and hazards.

In decision block 48, performance of users can be evaluated in anadaptive way in order to progress successive hazard scenes to moredifficult or challenging test conditions when the user exhibitssuccessful performance. An adaptive process helps ensure that the usercontinues to be challenged while avoiding frustration from havingextremely difficult test conditions. In order to provide a diverse setof available driving and hazard scenarios, a central server 50 isprovided which is in communication with simulation controller 40 via anetwork 51 (e.g., via the Internet). Central server 50 stores userprofiles in a profile database 52 and stores the scenarios in a database53. User profiles may include user achievement scores according to the“perception performance levels” compiled during simulations run in atesting mode. Based on the scores, corresponding path progressionsequences and hazard scenes can be selectably transmitted to simulationcontroller 40 from database 53.

FIG. 5 shows an embodiment of a user input device 55 to be used as asource of command signals generated manually by the user to performvarious driving actions and to control the simulation program.Directional elements on input device 55 such as a directional button pad56 and/or joystick elements 57 can be used to perform driving actionssuch as steering, acceleration, and braking. The directional elementscan also be used to perform a manual pointing function in which the useridentifies an object in a scene or in which the user navigates through acontrol menu for the overall application program. Pushbuttons 58 arealso provided in order to finalize a selection or to activate a menuitem, for example.

FIG. 6 depicts another preferred apparatus of the invention in greaterdetail, wherein a computing device 60 (such as a smartphone) is employedby a user 61 after inserting it into (or interconnecting it with) a VRheadset 62. Headset 62 is worn by the user 61 in the usual manner, and awireless controller 63 is held by user 61 to control a correspondingsimulation application program and to control a virtual vehicle in asimulated driving cycle. Computing device 60 functions as the simulationcontroller by executing a simulation program 64. In an input block 65,simulation program 64 processes user inputs received from a Bluetoothmodule 66 in communication with a Bluetooth module 67 in handheldcontroller 63 and from a motion sensor such as an accelerometer block 68in smartphone 60. Accelerometers 68 respond to head movement of user 61transmitted via headset 62. Handheld controller 63 includes an inputprocessor 70 coupled with button and joystick inputs 71, and theresulting command signals are transmitted between Bluetooth modules 67and 66.

During a simulated driving sequence, the user input command signalscorresponding to various driving actions are used in simulation program64 to generate an updated scene 72 according to continuously displayed3D animations simulating the driving sequence. In addition to generatinga stereoscopic display at 73, stereo audio signals 74 may be generatedto enhance realism of the simulation and to provide various feedbackaccording to the successful or the unsuccessful perception of hazardsduring operation of the simulation program. Smartphone 60 includes adisplay 75 for receiving the updated stereoscopic (left-right) displayanimations and a headphone port 76 for receiving the stereo audiosignals. Headset 62 includes lenses 77 for focusing the stereo displayto the right and left eyes of the user and headphone speakers 78 forreproducing the stereo audio signals.

FIG. 7 shows a preferred method in greater detail wherein, after beinglaunched, the simulation program accesses a user's profile and selectscorresponding path progression sequences and hazard scenarios in step80. Based on the virtual world containing the possible driving paths andhazard scenes to be depicted, a 3D environment for a particular drivingsequence is generated in step 81. The simulation begins in step 82 withthe user interactively navigating the virtual 3D environments based onuser commands entered using the handheld input device. In step 83, thesimulation controller starts a hazard scene at some unpredictablepredetermined time, wherein the hazard scene is comprised of a virtualobject representing a safety risk (typically corresponding to apotential collision based on the current relative velocities). A timeris started in step 84 at the moment when the virtual object isintroduced into the simulation.

A check is performed in step 85 for any user manifestations which wouldbe indicative of the user having perceived the hazardous virtual object.The manifestations can include any physical movement of the user whichhas been prompted by the appearance of the virtual object, such as thehead of the user turning toward the virtual object (i.e., turning of thehead to bring the virtual object toward the center of the user's fieldof vision) or a manipulation of the input device which matches apredetermined driving action that would be expected to evade the virtualobject such as a speed-reduction command (e.g., slowing down or stoppingthe virtual vehicle). Thus, the physical movement can just indicate theuser's perception of the potential safety risk or can correspond to theuser's reaction which is directed to an action to avoid the safety risk(implicitly establishing that the hazardous virtual object wasperceived).

In step 86, the timer that was started in step 84 is stopped at themoment when the first user manifestation is detected in step 85. Theresulting response time may then be ranked according to whether itachieves an acceptable performance level. Optionally, the invention canalso evaluate the correctness of an evasive maneuver taken by the user,if any. In step 88, a check is performed to determine whether thesimulation system is operating in a training mode. If so, then aninstruction phase 89 is conducted wherein the 3D animations replay thehazard scene while highlighting the virtual object which created thesafety risk. The appropriate reaction which should have taken can alsobe displayed if desired. After completing the instruction phase (orafter skipping the instruction phase if in a testing mode), then a checkis performed in step 90 to determine whether the current simulation iscomplete. If not, then the method returns to step 82 to continuenavigating through the current driving sequence. Otherwise, anyresulting performance metrics from a current testing mode are stored inthe user's profile in step 91.

What is claimed is:
 1. A virtual reality (VR) driving simulator forpresenting simulated hazard events to a user, comprising: a VR headsetmountable to a head of the user and having a display system forpresenting respective left and right images to left and right eyes ofthe user; a simulation controller coupled to the display systemgenerating 3D animations simulating a driving sequence; and a user inputdevice wirelessly coupled to the simulation controller for generating atleast one command signal corresponding to a driving action in responseto the 3D animations; wherein the 3D animations generated by thesimulation controller include a path progression sequence depicting avehicle drive cycle including driver actions defined by the commandsignal; wherein the 3D animations generated by the simulation controllerinclude a hazard scene introduced into the path progression sequence ata predetermined moment representing a safety risk which is dependentupon a user perception and a user reaction; wherein the simulationcontroller compares the user perception to predetermined perceptionperformance levels; wherein the simulation controller has a trainingmode wherein when the a user perception is detected below a selected oneof the performance levels then the hazard scene includes an instructionphase highlighting a source of the safety risk; and wherein thesimulation controller has a testing mode wherein the instruction phaseis not included.
 2. The driving simulator of claim 1 wherein the hazardscene comprises a virtual object with a relative location and trajectorycorresponding to a potential collision in the depicted vehicle drivecycle.
 3. The driving simulator of claim 2 wherein the user perceptionis measured as a time period from the predetermined moment until aphysical movement by the user prompted by the virtual object.
 4. Thedriving simulator of claim 3 further comprising a motion sensorproviding motion signals to the simulation controller and disposed inthe VR headset for detecting movements of the head of the user; whereinthe physical movement prompted by the virtual object and detected by thesimulation controller in response to the motion signals is comprised ofthe head of the user turning to bring the virtual object to a center ofa field of vision of the user or an eye movement of the user to bringthe virtual object to the center of the field of vision.
 5. The drivingsimulator of claim 3 wherein the physical movement prompted by thevirtual object and detected by the simulation controller in response tothe motion signals is comprised of a command signal generated by theuser input device corresponding to a predetermined driving action thatevades the virtual object.
 6. The driving simulator of claim 5 whereinthe driver actions defined by the command signal include a simulatedvehicle speed in the 3D animations, and wherein the predetermineddriving action is comprised of a speed-reduction command signal.
 7. Thedriving simulator of claim 5 wherein the driver actions defined by thecommand signal include a simulated vehicle speed in the 3D animations,and wherein the predetermined driving action is comprised of aspeed-reduction command signal followed by a user gesture indicating thevirtual object.
 8. The driving simulator of claim 7 further comprising amotion sensor providing motion signals to the simulation controller anddisposed in the VR headset for detecting movements of the head of theuser; wherein the user gesture indicating the virtual object iscomprised of the head of the user turning to bring the virtual object toa center of a field of vision of the user.
 9. The driving simulator ofclaim 7 wherein the user input device includes a button pad providing amanual pointing function, and wherein the user gesture indicating thevirtual object is comprised of manually pointing to an apparent positionof the virtual object.
 10. The driving simulator of claim 1 furthercomprising: a central server in wireless communication with thesimulation controller, wherein the central server stores a user profileincluding detected perception performance levels achieved by the userduring the testing mode.
 11. The driving simulator of claim 10 whereinthe central server stores a plurality of path progression sequences andhazard scenes which are selectably transmitted to the simulationcontroller in response to the perception performance levels achieved bythe user.
 12. The driving simulator of claim 1 wherein the simulationcontroller is comprised of a personal mobile device, wherein the VRheadset includes a compartment configured to receive the mobile device,and wherein the display system is comprised of a display screen of themobile device and a lens system of the VR headset.